Ugur HALICI - METU EEE - ANKARA 


11/18/2004 


ANN 



CAJAETSE J 

From Biological 
to Artificial Neuron Model 


ANN ee543 

CrLSP'/ER J ; From Biological to Artificial Neuron Model 


What 

you 

see 

in 

the 

picture? 



EE543 - ANN - CHAPTER 1 


1 


Ugur HALICI - METU EEE - ANKARA 


11/18/2004 


ANN ee543 

J ; From Biological to Artificial Neuron Model 



■ Is there any conventional computer at 
present with the capability of perceiving 
both the trees and Baker's transparent 
head in this picture at the same time? 

* Most probably, the answer is no. 

■ Although such a visual perception is an 
easy task for human being, we are faced 
with difficulties when sequential 
computers are to be programmed to 
perform visual operations. 
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Conventional Computers 

■ In a conventional computer, usually there exist a single processor 
implementing a sequence of arithmetic and logical operations, 
nowadays at speeds approaching 10 9 operations per second. 

i However this type of devices have ability neither to adapt their 
structure nor to learn in the way that human being does. 
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What traditional computers can ’t do? 

■ There is a large number of tasks for which it is proved to be virtually 
impossible to device an algorithm or sequence of arithmetic and/or 
logical operations. 

> For example, in spite of many attempts, a machine has not yet been 
produced which will automatically read handwritten characters, or 
recognize words spoken by any speaker let alone translate from one 
language to another, or drive a car, or walk and run as an animal or 
human being [Hecht-Nielsen 88] 
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Computers versus Brain 

i What makes such a difference brain an conventional computers seems 
to be neither because of the processing speed of the computers 
nor because of their processing ability. 

■ Today’s processors have a speed 10 5 times faster than the basic 
processing elements of the brain called neuron. 

■ When the abilities are compared, the neurons are much simpler. 

• The difference is mainly due to the structural and operational trend. 

• While in a conventional computer the instructions are executed 
sequentially in a complicated and fast processor, the brain is a massively 
parallel interconnection of relatively simple and slow processing 
elements. 
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1.1. Biological Neuron Human nervous system 

■ It is claimed that the human central nervous system is comprised of 

about 1,3x1 0 10 neurons and that about 1x1 0 10 of them takes 
place in the brain. 

■ At any time, some of these neurons are firing and the power dissipation 

due this electrical activity is estimated to be in the order of 10 
watts. 

> Monitoring the activity in the brain has shown that, even when asleep, 
5x1 0 7 nerve impulses per second are being relayed back and forth 
between the brain and other parts of the body. This rate is 
increased significantly when awake [Fischer 1987], 
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1.1. Biological Neuron Neuron 

i A neuron has a roughly spherical cell body called soma (Figure 1.1). 
i The signals generated in soma are transmitted to other neurons through 
an extension on the cell body called axon or nerve fibres. 
i Another kind of extensions around the cell body like bushy tree is the 
dendrites, which are responsible from receiving the incoming 
signals generated by other neurons. [Noakes 92] 
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1.1. Biological Neuron Axon 

* An axon (Figure 1.2), having a length varying from a fraction of a 
millimeter to a meter in human body, prolongs from the cell body at 
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1.1. Biological Neuron Sy napses 

■ At the other end, the axon is separated into several b 

end of which the axon enlarges and forms term 

■ Terminal buttons are placed in special structures c 

which are the junctions transmitting signals 1 
another (Figure 1 .3). 
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Figure 1.3. The synapse 
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1.1. Biological Neuron Synapses 

■ A neuron typically drive 10 3 to 10 4 synaptic junctions 

■ The synaptic vesicles holding several thousands of molecules of 

Chemical transmitters, take place in terminal buttons. 

■ When a nerve impulse arrives at the synapse, some of these Chemical 

transmitters are discharged into synaptic cleft, which is the narrow 
gap between the terminal button of the neuron transmitting the 
signal and the membrane of the neuron receiving it. 

■ In general the synapses take place between an axon branch of a neuron 

and the dendrite of another one. Although it is not very common, 
synapses may also take place between two axons or two dendrites 
of different cells or between an axon and a cell body. 
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1.1. Biological Neuron Ion pumps 

i Neurons are covered with a semi-permeable membrane, with only 5 
nanometer thickness. 

i The membrane is able to selectively absorb and reject ions in the 
intracellular fluid. The membrane basically acts as an ion pump to 
maintain a different ion concentration between the intracellular 
fluid and extracellular fluid. 

i While the sodium ions are continually removed from the intracellular fluid 
to extracellular fluid, the potassium ions are absorbed from the 
extracellular fluid in order to maintain an equilibrium condition. 
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1.1. Biological Neuron Resting Potential 

■ Due to the difference in the ion concentrations inside and outside, the 

cell membrane become polarized. 

■ In equilibrium the interior of the cell is observed to be 70 milivolts 

negative with respect to the outside of the cell. The mentioned 
potential is called the resting potential. 
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1.1. Biological Neuron graded potential 

■ A neuron receives inputs from a large number of neurons via its synaptic 

connections. Nerve signals arriving at the presynaptic cell 
membrane cause Chemical transmitters to be released in to the 
synaptic cleft. 

i These Chemical transmitters diffuse across the gap and join to the 
postsynaptic membrane of the receptor site. The membrane of the 
post-synaptic cell gathers the Chemical transmitters. 

■ This causes either a decrease or an increase in the efficiency of the 

local sodium and potassium pumps depending on the type of the 
Chemicals released in to the synaptic cleft. 
l In turn, the soma potential, which is called graded potential, changes. 
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1.1. Biological Neuron Excitatory, Inhibitory synapses 

■ While the synapses whose activation decreases the efficiency of the 

pumps cause depolarization of the graded potential, the effects of 
the synapses that increase the efficiency of pumps result in 
hyperpolarization. 

■ The first kind of synapses encouraging depolarization is called 

excitatory and the others discouraging it are called inhibitory 
synapses. 
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1.1. Biological Neuron Firing 

■ If the decrease in the neuron polarization is adequate to exceed a 

threshold then the post-synaptic neuron fires. 

■ The arrival of impulses to excitatory synapses adds to the depolarization 

of soma while inhibitory effect tends to cancel out the depolarizing 
effect of excitatory impulse. 

* In general, although the depolarization due to a single synapse is not 
enough to fire the neuron, if some other areas of the membrane 
are depolarized at the same time by the arrival of nerve impulses 
through other synapses, it may be adequate to exceed the 
threshold and fire. 
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1.1. Biological Neuron Action Potential 

* At the axon hillock, the excitatory effects result in the interruption the 
regular ion transportation through the cell membrane, so that the 
ionic concentrations immediately begin to equalize as ions 
diffuse through the membrane. 



Figure 1.4. The action potential on axon 
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1.1. Biological Neuron Action Potential 

i If the depolarization is large enough, the membrane potential at axon 
hillock eventually collapses, and for a short period of time the 
internal potential becomes positive. 

i The action potential is the name of this brief reversal in the potential, 
which results in an electric current flowing from the region at action 
potential to an adjacent axon region with a resting potential. 

• This current causes the potential of the next resting region to change, so 
the effect propagates in this manner along the axon membrane 
wall. 


EE543 - ANN - CHAPTER 1 


9 



Ugur HALICI - METU EEE - ANKARA 


11/18/2004 


ANN ee543 

CrJAPT.SE J i From Biologkul tø Artificial Neuron Model 


1.1. Biological Neuron Refractory Period 

■ Once an action potential has passed a given point, it is incapable of 

being reexcited for a while called refractory period. 

■ Because the depolarized parts of the neuron are in a State of recovery 

and can not immediately become active again, the pulse of 
electrical activity always propagates in only forward direction. 

■ The previously triggered region then rapidly recovers to the polarized 

resting State due to the action of the sodium potassium pumps. 

> The refractory period is about 1 milliseconds, and this limits the nerve 
pulse transmission so that a neuron can typically fire and generate 
nerve pulses at a rate up to 1000 pulses per second. 
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1.1. Biological Neuron Propagation of signals on axon 

■ The number of impulses and the speed at which they arrive at the 

synaptic junctions to a particular neuron determine whether the 
total excitatory depolarization is sufficient to cause the neuron to 
fire and so to send a nerve impulse down its axon. 

■ The depolarization effect can propagate along the cell wall but these 

effects can be dissipated before reaching the axon. 

■ However, once the nerve impulse created at axon hillock it will 

propagate in axon until it reaches the synapses where the 
depolarization effect will cause the release of Chemical 
transmitters into the synaptic cleft. 
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1.1. Biological Neuron pulses on axon 
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1.1. Biological Neuron Myelin sheath and nodes of Ranvier 

i The axons are generally enclosed by myelin sheath that is made of 
many layers of Schwann cells promoting the growth of the axon. 

i The speed of propagation down the axon depends on the thickness of 
the myelin sheath that provides for the insulation of the axon from 
the extracellular fluid and prevents the transmission of ions across 
the membrane. 

i The myelin sheath is interrupted at regular intervals by narrow gaps 
called nodes of Ranvier where extracellular fluid makes contact 
with membrane and the transfer of ions occur. 
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1.1. Biological Neuron Myelin sheath and nodes of Ranvier 

■ Since the myelin sheath is poor conductor, the action potential is 

transmitted as depolarizations occur at the nodes of Ranvier. 

■ This happens in a sequential manner so that the depolarization of a 

node triggers the depolarization of the next one. 

■ The nerve impulse effectively jumps from a node to the next one along 

the axon each node acting rather like a regeneration amplifier to 
compensate for losses. 

■ Once an action potential is created at the axon hillock, it is transmitted 

through the axon to other neurons. 
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1.1. Biological Neuron The nature of signal transmission in the ns 

i It is mostly tempted to conclude the signal transmission in the nervous 
system as having a digital nature in which a neuron is assumed to 
be either fully active or inactive. 

i However this conclusion is not that correct, because the intensity of a 
neuron signal is coded in the frequency of pulses. 

l A better conclusion would be to interpret the biological neural systems 
as if using a form of pulse frequency modulation to transmit 
information. 
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1.1. Biological Neuron The nature of signal transmission in the ns 

■ The nerve pulses passing along the axon of a particular neuron are of 

approximately constant amplitude but the number generated 
pulses and their time spacing is controlled by the statistics 
associated with the arrival at the neuron's many synaptic junctions 
of sufficient excitatory inputs [Muller and Reinhardt 90], 

■ The representation of biophysical neuron output behavior is shown 

schematically in Figure 1.5. [Kandel 85, Sejnowski 81] 
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1.1. Biological Neuron The nature of signal transmission in the ns 

i At time t=0 a neuron is excited; at time T, typically it may be of the order 
of 50 milliseconds, the neuron fires a train of impulses along its 
axon. Each of these impulses is practically of identical amplitude, 
i Some time later, say around t=T+x, the neuron may fire another train of 
impulses, as a result of the same excitation, though the second 
train of impulses will usually contain a smaller number. Even when 
the neuron is not excited, it may send out impulses at random, 
though much less frequently than the case when it is excited. 
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Figure 1.5. Representation of biophysical neuron output signal 
after excitation at tine t=0 
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1.2. Artificial Neuron Model 

■ As it is mentioned in the previous section, the transmission of a signal 
from one neuron to another through synapses is a complex 
Chemical process in which specific transmitter substances are 
released from the sending side of the junction. 

i The effect is to raise or lower the electrical potential inside the body of 
the receiving cell. If this potential reaches a threshold, the neuron 
fires. 

l It is this characteristic that the artificial neuron model proposed by 
McCulloch and Pitts, [McCulloch and Pitts 1943] attempt to 
reproduce. 
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1.2. Artificial Neuron Model 



Figur e 1.6. Artificial Neuron 

■ The neuron model shown in Figure 1.6 is the one that widely used in 
artificial neural networks with some minor modifications on it. 

■ It has N input, denoted as u v u 2 , ...u N . 

I Each line connecting these inputs to the neuron is assigned a weight, 
which are denoted as w v w 2 , w N respectively. Weights in the 
artificial model correspond to the synaptic connections in biological 
neurons. 

i 0 represent threshold in artificial neuron. 

I The inputs and the weights are real values. 
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1.2. Artificial Neuron Model Neuron Activation 
i The activation is given by the formula: 

N 

a = (£wjUj) + e ( 1 . 2 . 1 ) 

7=1 

i A negative value for a weight indicates an inhibitory connection while a 
positive value indicates an excitatory one. 
i Although in biological neurons, 0 has a negative value, it may be 
assigned a positive value in artificial neuron models. If 0 is 
positive, it is usually referred as bias. For its mathematical 
convenience we will use (+) sign in the activation formula. 
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1.2. Artificial Neuron Model Neuron Activation 

■ Sometimes, the threshold is combined for simplicity into the summation 
part by assuming an imaginary input u 0 =+1 and a connection 
weight w 0 = 0. Hence the activation formula becomes: 


N 

a =H w j u j (1.2.2) 

7=o 
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1.2. Artificial Neuron Model Neuron Activation 
i The vector notation 


is useful for expressing the activation for a neuron. 

■ Here, the f' element of the input vector u is t^and the f h element of the 
weight vector of w is w y Both of these vectors are of size N. 

I Notice that, w T u is the inner product of the vectors w and u, resulting in 
a scalar value. The inner product is an operation defined on equal 
sized vectors. In the case these vectors have unit length, the inner 
product is a measure of similarity of these vectors. 
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1.2. Artificial Neuron Model Neuron output 

l The output value of the neuron is a function of its activation in an 
analogy to the firing frequency of the biological neurons: 

* = /(«) (1.2.4) 

l Originally the neuron output function f(a) in McCulloch Pitts model 
proposed as threshold function, however linear, ramp and sigmoid 
functions (Figure 1 .7.) are also widely used output functions. 
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1.2. Artificial Neuron Model Neuron output function 



Figur e 1. 7. Some neuron output functions; 

a) threshold function 

b) ramp function, 

c) sigmoid function, 

d) Gaussian function 
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1.2. Artificial Neuron Model Neuron output 
Linear: f( a ) = m 

(1.2.5) 

Threshold: 

f(a) = l° a< ° 
\l 0 <a 

(1.2.6) 

Ramp: 

l0 a< 0 

(1.2.7) 

Sigmoid: 

f(a) = la/ic 0 <a<K 

\1 K<a 

fid) = 1 

\ + e~ Ka 

(1.2.8) 
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1.2. Artificial Neuron Model Implementation of If by an 

i Though its simple structure, McCulloch-Pitts neuron is a powerful 
computational device. 

i McCulloch and Pitts proved that a synchronous assembly of such 
neurons is capable in principle to perform any computation that an 
ordinary digital computer can, though not necessarily so rapidly or 
conveniently [Hertz et al 91]. 
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1.2. Artificial Neuron Model Example 1.1 

When the threshold function is used as the neuron output function, and 
binary input values 0 and 1 are assumed, the basic Boolean functions 
AND, OR and NOT of two variables can be implemented by choosing 
appropriate weights and threshold values, as shown in Figure 1.8. 
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1.2. Artificial Neuron Model Example 1.1 



Figure 1.8. Implementation ofBoolean functions by artificial neuron 
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1.3. NetWork of Neurons 

■ While a single artificial neuron is not be able to implement some boolean 

functions, the problem is overcome by connecting the outputs of 
some neurons as input to the others, so constituting a neural 
network. 

■ Suppose that we have connected many artificial neurons that we 

introduced in Section 1.2 to form a network. In such a case, there 
are several neurons in the system, so we assign indices to the 
neurons to discriminate between them. 
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1.3. NetWork of Neurons 


W Then to express the activation or output of i th neuron, the formulas are 
modified as follows: 

./=! 


where Xj may be either the output of a neuron determined as 


•V, ./'(«;) 


(1.3.2) 


or an external input determined as: 

Xj=Uj (1 .3.3) 
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1.3. NetWork of Neurons 

■ In some applications the threshold value 0/ is determined by the external 
inputs. 

I Due to the equation (1.3.1) sometimes it may be convenient to think all 
the inputs are connected to the network only through the threshold 
of some special neurons called the input neurons. They are just 
conveying the input value connected to their threshold as 0/= uj to 
their output xj with a linear output transfer function fj(a)=a. 
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1.3. NetWork of Neurons 


• For a neural network we can define a State vector x in which the / th 

component is the output of ith neuron, that is x r 

• Furthermore we define a weight matrix W, in which the component 

Wji is the weight of the connection from neuron j to neuron i. 

• Therefore we can represent the system as: 

x = f(W T x + ø) (1.3.4) 

Here 0 is the vector whose i th component is 0, and f is used to 
denote the vector function such that the function f) is applied at the 
z th component of the vector. 
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1.3. Network of Neurons 

Example 1.2: A simple example often given to illustrate the behavior of a 
neural networks is the one used to implement the XOR (exclusive OR) 
function. Notice that it is not possible to obtain exclusive-or or equivalence 
function, by using a single neuron. However this function can be obtained 
when outputs of some neurons are connected as inputs to some other 
neurons. Such a function can be obtained in several ways, only two of 
them being shown in Figure 1.9. 



Figur e 1.9. Various implementations of the exclusive-or function 
by using artificial neuron 
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1.3. NetWork of Neurons 

The second neural network of Figure 1 .9 can be represented as: 


t 




where f\ and f 2 being linear identity function, f and f 4 being threshold 
functions. 

In the case of binary input, u t e{0,1} or bipolar input, that is Uj e{-1,1}, all of f 
may be chosen as threshold function. 

The diagonal entries of the weight matrix are zero, since the neurons do not 
have self-feedback in this example. 

The weight matrix is upper triangular, since the network is feedforward. 
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1.4. Network Ar chitectur es 

■ Neural computing is an alternative to programmed computing which is a 

mathematical model inspired by biological models. 

I This computing system is made up of a number of artificial neurons and 
a huge number of interconnections between them. 

■ According to the structure of the connections, we identify different 

classes of network architectures (Figure 1.10). 



Figure 1.10 a) layered feedforward neural network 
b) nonlayered recurrent neural network 
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1.4. NetWork Ar chitectures Feedforward Neural networks 


i In feedforward neural networks, the 
neurons are organized in the form of 
layers. 

■ The neurons in a layer get input from the 
previous layer and feed their 
output to the next layer. In this kind of 
networks connections to the neurons 
in the same or previous layers are not 
permitted. 

l The last layer of neurons is called the 
output layer and the layers between 
the input and output layers are called 
the hidden layers. 

l The input layer is made up of special input 
neurons, transmitting only the applied 
external input to their outputs. 
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1.4. NetWork Ar chitectures Feedforward Neural networks 

i In a network, if there is only the layer of input nodes and a single layer 
of neurons constituting the output layer then they are 
called single layer network. 

■ If there are one or more hidden layers, such networks are called 

multilayer networks. 

■ For a feed-forward network always exists an assignment of indices to 

neurons resulting in a triangular weight matrix. Furthermore if 
the diagonal entries are zero this indicates that there is no 
self-feedback on the neurons. 
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1.4. NetWork Ar chitectures Recurrent Neur al Networks 


■ The structures, in which connections to 

the neurons of the same layer or to 

the previous layers are allowed, are called 
recurrent networks. 

■ In recurrent networks, due to feedback, it 
is not possible to obtain a triangular 
weight matrix with any assignment of the 
indices. 
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